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In two experimental studies, tnKef f ectiveness of 
' teaching skills are examined in the context of: (xK^a specific 

teaching method linked to (2) specific curriculum objectives and (3) 
specific curriculum materials; the method and materials used are by 
(U) specific teachers and (5) specific students, in a (6) specific 
instructional setting; finally, (7) tests are created that are 
responsive to the idiosyncratic outcomes of the teaching skills and 
curriculum objectives. Problems and possible solutions involved in 
creating such a context for studying teaching skills are examined. By 
referring to these experiments on how the questioning skills of ^ 
sixth-grade teachers affect student learning outcomes, an example is 
provided of how methodological ' requirements are met with respect to 
each of the seven context varia^bles. Each context variable is 
considered separately, and the three following methodological 
requirements are satisfied: (1) ecological validity, the creation of 
each aspect of the experimental aonte^t so that it reflects real 
classroom teaching; (2) fidelity of treatment, a close correspondence 
between intended context and what actually occurs experimentally; and 
(3) achievement of eocparimental control, either by holding context 
variables constant or by allowing them to vary randomly. A major 
problem cited is the expense of conducting the experiment since 
resources were not already available to design and run teaching 
skills experiments on this scale. It was decided that in the future 
research costs can be reduced/ by designing experiments that use 
instructional context developed in previous studies. (MM) 
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TUe Impc^rtaiict' ot uuntext Variaoles in Kesearch on Teaching Skills^ 

Meredith D. Gall 
university ot Oregon*^ 

iiow effective are teacninv; skills — for example, the use of verbal praise, 
aci^nowledj^raent of btudent. laeas, and hieiicr cognitive questions? Do teachers who 
use these skills promote better learninj^ outcomes in their students than do teachers 
tpv vho use these skills to a lesser dek;ree, or not at all? There has been much interest 
^ recently in doing experimental studies to determine the effectiveness of teaching 
skills (Dunkin and biddle, 1974). 

1^ In doing these studie.s, re.searcher.^ need to take into account the fact that te^aching 
^ ^ skills do not exist in a vacuom. ihev must be studied in the context of a specific 
t eaching raethod ; the nethod must be linked to specific curriculum objectives and 
perhaps to specific curriculum na^terials . Furthermore, the method and materials 
VX,' must be used by specific tea chers and students in a specific instructional setting * 
Finally, tests must be created which are responsive to the idiosyncratic outcomes 
of the teaching skills and curriculum objectives. 

It a meaniniL^ful context is not created, researchers will have difftculty inter- 
preting the results of exper L'^'^nts on teaching skills. The purpose of this paper 
is to describe problems and possible solutions involved in creating a context in 
which teaching skills can oe i3.tudied e'^periraentally. Each context variable will 
be considered separately: teachint: method, curriculum objectives and materials, 
teachers, students, instructional setting, and testing procedures. 

ihree metaodoio^icai requirenents must be satisfied in creating each aspect of the 
experimental context. First is ecological validity, that is, the creation of each 
aspect of tne experimental context so that it reflects "real" classroom teaching. 
The second requl^*eraent is fidelity of treatment, that is, a* close correspondence 
between the intended context and what actually occurred when the experiqient was 
'run." hie tnird methodological requirement is to achieve experimental control 
eitner oy holding context variables constant or by allowing them to vary randomly. 

i'ne methodological reqairements witii respert to each context variable can be analyzed 
by referrlu.; to two experiments on quest lonin*', skills conducted by the author and 
his coilea^ues (Gall e_t al., l>7b). T\h^ purpose of the two experiments was to deter- 
mine student learning outcomes which are affected by sixth-erade teachers' use of 
certain questioning SKills. In the first experiment teachers conducted discussions 
which differed in the presence or absence of two questioniui; skills: probing and 
redirection. Probing occurs when tae teacher seeks to improve the quality of a 
student's initial answer to a question bv asking a follow-up (that is, a "probing'*) 
question. Redirection occurs when the teacher calls upon more than one student 
to respond to a question already asked of another student. It was hypothesized 
that probint; and redirection would promote learning since they provide students with 
practice in orp,,ini2ing their facts and ideas into overt responses. 

In the &econd experirr^nt t»=*acuers conducted discussions which differed in percentage 
of higher cognitive questions (J3^>, 3t)/>, or A higher cognitive question was 

defined according to two criteria: first, it must require the student to state 
predictions, solutions, explanations, evidence, ^.eneralizations , interpretations, 
or opinions. The second criterion is that the predJction, solution, etc. asked for 
in the question must not be directly available in the curriculum materials. It was 
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hypothesized ttu\t disc us>>iuH3 witii a hik;h percenta^'C -jf these questions would 
promote nore leaniin>; than discussions with a low percentage of these questions. 

^chin^, Method 

fhe questioninjt' skills in the two experiments were studied in the context of the 
discussion method. Originally the research team considered creating systematic 
variation by asking some teachers to conduct discussions in which they made use 
ot certain of the skills, such as probing and redirection. Another group of 
teachers (or possibly the sane croup of teachers) would be asked to conduct dis- 
cussions in which theyomitted use of these skills. 

Ihis approach creates several problems. First, many features of the discussion 
which mii^ht affect learning outcomes are left uncontrolled. For example, some 
teachers mi>;ht ask more questions related to tl.e curriculum objectives than other 
teachers mjlght ask; or thev simply mit^ht ask a greater number of questions. As 
a result, some students might have more "opportunity to learn" than others. If 
teachers; in the experimental group (use of probing and redirection) happened to 
ask more questions not related to the curriculum than the control teachers 
(absence of probing and redirection), the potential positive effects of the skills 
might be cancelled. 

Anotner possible probleni with tliii> approach is that teachers within each treatment 
group mi,:ht Implement their treatrnent to varying dei;rees. For example, some 
teachers in the experimental group might fail to use probing and redirection, 
whereas some teachers in the control group might use the^^e skills inadvertently. 
U this phenomenon occurs, it uecomes very ditficult to determine just what the 
V. ' treatment" was . / 

These problems create a need tor experimental control of the teaching method in 
which the skills are emheda«_>d. : he solution developed bv the author and his 
colleav;uo3 was to create a tormat desi^inated "seni-programmed discussion." The 
discussionb were programnied in that they were scripted in advance by the research 
team. The teachers were instructed to follow the scripts exactly. Each scripted 
discussion contained sixteen questions in a f ixeu sequence. This aspect of the 
discussion rietnod could be ^^.pecified rather exactly, thus insuring uniformity of 
treatment across teachers, however, probing and redirection could not be scripted 
to the sane degree. For example, if a student whom the teacher called upon to 
answer a aigtier cognitive questiua gave a good anwer, the teacher would have 
little justification to probe. In contrast, a weak answer would justify probing, 
because of this problem, teachers were trained to use general guidelines rather 
than exact scripts with respect to probing and redirection. Hence, the designa- 
tion, "semi-programmed*' aiscussions. 

ihe stmi-programmea format tias tue auvanta/.e of experimental control, but it is 
unrepre*c>entat ive (low ecological validity) of classroom teaching since teachers 
typically do not structure discussions to tnis extent, liowever, the researchers 
attempted to build representativeness into other features of the experimental 
discussions. For example, pilot work by the research team indicated that sixth- 
grade teachers typically conduct ».iiscubsions of 20 to 30 minute duration, during 
which tuey ask fifteen to twenty substantive questions. The semi-programmed 
discussions contained sixteen questions in order to represent the natural setting. 
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The scripts were t icl<i-t tested to assess the fe.isibilltv of \he semi-propTanuned 
forraac. Results of the field test indicatrd that teachers ,|.if trained, could 
follow the scripts in a natural nanner. Thr probinr, ana redirection treatment 
.generated substantial disoussion tor iaost of the questions, t!ie no probinr and 
redirection treatment coula he liken^^d to a simple cuestion-and-answer format. 
Sone teachers comniented tt»ac the scripts facilitated the teaching task: since 
they knew in advance the rjue^tions to be abked , they could devote more attention 
to the students' behavior. The research team and other persons listened to 
audiotapes of the semi-programmed discussions. The discussions appeared natural 
and realistic to them. At the conclusion of the experiment, students ^lenerally 
exprebsed positive attitudes on a Likert-tvne scale toward the semi-programmed 
discussions . 

These results demonstrate ti^e feasibilltv of usin^ a proc^rammin^ technique to 
cichieve experimental coutroi over teaching method as a context variable, while 
^t the ':Jame time preserving representativeness. 

Anotlier advantav»e of the semi-pro.irammed format is tnat it promotes fidelity 
of treatment. In the tvo experiments on questionlnj^ skills, teachers were asked 
to audiotape several of their discussions. The sampling procedure was used. ^ 
instead of audiotapinv; all dliicusslons because teachers found the procedure 
somewhat disruptive. Raters were employed to score the tapes on variables 
considered to be critical to treatment fidelity. For example, the raters checked 
each discussion to determine whether the teachers (a) asked or omitted each 
scripted question, (b) made substantial chanfes in the wording of a question, and 
(c) asked the questions in a different order than was presented in the script. 
Results of this analysis indicated tmr deviations from the scripts were infrequent. 
The most deviance occurred in the order of question-askinr . Deviations were 
observed in only six of the 14A mdiotaped discussions. 

The raters oDserved frequency of probing and redirection in each treatment group. 
As intended, teachers in the prooxn^ and redirection treatment followed the 
guidelines oy using' these skUIs with l.i^.. <^"requencv: prohinR (X « 21.45) and 
redirection (-'^ ^ 2.2. ^b). In the no probinc and redirection treatment, teachers 
were instructed to avoid use of these skills. Audiotape analys^.s revealed high, 
though i^ot perfect, fidelity of treatment: proline (X = 3.29) and redirection 
(a = O.iJ). Other analyses also revealed hit;h fidelity of treatment. 

In summary » it appears tiiat the semi-propranuned format is very useful in helping 
teachers, first, to use the teachin^^ method intended by the researchers, and, 
second, to make a quantitative check of the correspondence between the researchers* 
^ Intent and the teacher's behavior, that is, fidelity of treatment. 

Curr iculu m M aterials and Objectives 

Curriculum objectives and materials lorm' another important set of context variables 
wliich sb.oula be controiieu m researcli on teachinc; skills. In the author's exper- 
iments, questioning', skills were studitii in the context of the discussion method, 
which in turn was used in conjunction with a specially prepared set of curriculum 
materials. Lach ex[>er inental lesson consisted of a scripted discussion preceded 
by a brief (approximatel v 13 minutes) curriculum selection which students viewed 
and/or read. The same curriculun materials were riven to fill students in the 
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experiment, thore:>y i*,Mitia opportunlcv to learn the curriculum across treatment 
conditions. / 

/ 

lae currtculun materials were cor.ntructed to neet the requirements of experimental 
control and ecoIok;ical vaiialiy, that is, repre >cntati/eness with respect to 
clai^broom teacriint;. iiie latter ^oal was accomplished bv including several features 
in the materials. Fir^-t, tue curriculuiu consisted ot ten lessons, one per school 
day, tor a total of approxiiuateiv ten hours of instruction. This length is 
tvpical of curriculum units at the sixtii-^^rade level. Hy having a curriculum 
unit of this duration, ttie experiments avoided a :riticisn sometimes directed 
at other research on teaciiin/. skills, namely, that the total teaching/learning 
episode wa^ too short (some tines as short as 30 minutes) for effects of the 
teachinti skills to be observed. 

iiie second represent at i 'e feature of the curriculum unit was that it provided 
instruction in ecology, a topic which was perceived by the participating schools 
as relevant ana consistent with their objectives. Ihus, the curriculum was 
coraplementarv to the reit^ular school cirriculun, rather than an irrelevant intru- 
sion, ihird, the ecolo^'v piaterials were constructed bv following a typical 
curriculum development sequenne. Object Ives for the unit were identified; next 
readin**, i>elections, auaio-visuai^ , and scripted discussions were developed to 
help students acnieve the oi?jectives. Finally, the various components were field- 
tested and revised to ^et the "bu^is" out. This process was designed to insure 
tiiat the qaestioViin)?, bkills were used in the context of an effective curriculum, 
'icnervise the inef f ect Ivenet^s of the curriculum nic;!it ' obscure potential positive 
effects of the questioning skills under investi^r-.ation. 

I'ne fourth representative teature of the .curriculum is that it included a variety 
of instructional inateriais ovt;r t.ie duration oi the unit — an ecology game, a 
film, fiimstrips, and lilustratiM readme selections. Variety, which Itself has 
been s;iown to ue effective (:a'stMistilne and Kurst, lv71) contributed to establishing 
,t ^tr(Hi», currKuium context m wiich to invest ii;ate the questioning skills. The 
tittn feature is that ttie reauiu,; level oi tue materials was kept at or below 
tiie stu.:ents' .Tade-- level liiat re^Miii/! skill would not be a major influence 
on f^ielr ability to partiv.pat in the discussions associated with each treatment 
condition. Finally, Mie nattri.iLs were presented in an attractive format to 
stimulate and aiaintam students' interest. 

Fidelity oi treatment witti respect to curriculura materials was reJatively easy 
to observe, leacriers were lamlliarized wicl^ the materials and trained in their 
use prior to the uata collection phase. Jurin^ the data collection phase j the 
researcn team visited tiie teachers* classrooms to detemine whether students 
actually received the rr^aterials as intended. Fidelitv of this aspect of the 
treatment vas hi^h. In a f^w instances a teacher forgot tc bring the lesson 
nateriais to ciass. Mien tui^ uccurro^i, the k»sson was delayed until the research 
team monitoring', the eA])errent could proviiie a hack-up set of materials. 

In -iamraarv, it appears j:ea:,iMe to create a curriculuni context which is repre- 
sentative oi" actual classroo"-. teachin^^ and which is implemented as planned. 

i t'acii5r_. 

leacjir.- ^'►-ilis oovinusly cinnot he studied unles*^ teachers are available to 



iirplement t*»en. *tiu>, icacucrs become an inportant part of the context in wlrich 
teaching: skills are researched. Furthernore, teachers require trainini^ if the 
skills are to be used in the manner desired by the researchers. In the experiments 
on questioning skills, the training task was sir*plifieti b% the fact that teachers 
needed only to learn iiow to use prepared curriculum materials hnd a scripted 
discussion format. They did not need to v;enerate their own lesson plans and 
materials. ♦ , 

Despite this fact, the experiment still required five days of teacher training 
initially plus frequent refresher and review sessions throughout the four-week 
duration of tue cyrriculum units. ^ The decision was made not to use the students' 
regular teacners, Decau e they would find it difficult to attend the training 
sessions, .\nother reason for not using the regular teachers was that the exper- 
imental design (a latin square) required four different teachers to work at the 
same time with a particular class, each teacher administered a different treatment 
to approximately one-fourth of the students in the class. 

because of tiie requirements inposed by the research design, a separate group 
of teachers was trained to impleir.ent the treatments. Substitute teachers in a 
nearby sciiool district were recruited for this purpose. The primary criteria 
for selection as a teacner in the experiments were:(l) a California teaching 
credential; (2) experience in teaching elementary school children; and (3) 
willingness to contribute to a research i^roject. The first two criteria helped 
to insure that the experimental teachers would be representative of regular 
classroom teacners. After the initial training sessions, which occurred in 
schools similar to those used in the experiments, the teachers who were best at 
following the semi-pro i>:ranirted discussion format and at interacting with children 
were chosen to conduct the experiments. The success of this procedure was shown 
by the fact that the experimental teachers were well-accepted by the school 
'administrators and the regular teachers. There was not a single complaint or 
instance when the exper iuientai treatments were compromised because of friction 
between school personnel and research personnel. 

The use of special_teacher^ also contributed tt fidelity of treatment. Because 
the experimental "teacher^ were paid dlrectlv by tne research team, they could be 
held accountable for meetinr; tiie requirements of the treatment conditions. The 
same level of accountability would have been much more difficult to require of 
regular teachers. 

The experimental teacners responded well to the need for accountability. None 
of them resigned from the project until the experiments had been completed. They 
maintained a high level of fidelity to the aiscussion scripts, as described earlier 
in this paper. Furtnerr.ore , the data analyses showed that teacher effects accounted 
for very little of the variance in the dependent measures, even though the experi- 
mental aesi,;n required each teacuer to implement four different treatment conditions. 
Thus» it rppearo that teachers can be trained to conform to fairly precise standards 
of teaching performance. 

S tuden ts 

Students obviously are anotticr ir.portant aspect of the context in which teaching 
skills are used. To invest i^^ate questioning skills, especially the skill of asking 
higher cognitive questions, the research team needetl to define a group of students 
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who were repre&entactv*.^ ui the studentb with whom these skills are used. Sixth- 
grade students were chosen for ttiis reat>on, Discussions involving; probing, redirec- 
tion, anU Higher cognitive ^^ue^tions are conducttni \. 1th .students in the intermediate 
(fourth tltroui'Ji sixth) Kr^dfS. Another reason for selecting .students at this ^^rade 
level is ihat reading materials coulJ used to present much of the facts and ideas 
on which the i>emi-prOe;rannt;a discussi'-n-. were based. Substantially more effort by 
tlie researcn team would have been required to create curriculum materials which did 
not involve reading. 

Ihe most critical fidelity of treatment issue with "respect to students is whether 
they received their assir>ned treatment condition. One aspect of this issue is 
whether students actually attended each treatment session. In the experiments 
on questioning, attendance was recorded daily and was found to be quite hij*h. 
Absences occurred only when a student wa3 not in school, usually because of illnesst 
ochool personnel cooperated wit!* the research by not scheduling individual children 
for other activities, sucn as band practice, durlnt the treatment sessions. In 
a few instances an activity affecting the entire class was scheduled at the same 
time ay the treatment sessions. Ulien this t.ccurred, the treatment sessions simply 
were rescheduled. 

Lven when students attend trieir assigned treatment sessions, it is possible that 
they may not '"receive" the treatment. Students' physical presence does not mean 
necessarily that they are attending; to and processing the curriculum content. Lack 
of attention* is likely to occur if students misbehave- or are distracted by another 
activity. A shortcoming of the experiments on questioning skills is that this 
aspect of fidelity of treatment — students* atteiitiveness — was not monitored 
closfely. however, informal observation b\ the research team indicated that the 
students (generally were attentive tc- the curriculum materials and to the discussion 
process . 

Instru c tional Se tt in^^^ 

In designing tne expt;riraents on questioning skills, the research team considered 
the alternative of bringing students to their institute for the treatment sessions. 
This procedure would have made it easier to maintain certain experimental controls. 
However, the institute tiad the disadvantage of beinj> unrepresentative of the 
students* typical learning environment. An unfamiliar setting might mask or other- 
wise alter the effects of the questioning skills being investigated. Therefore, 
the experiments were conducted at the students* own schools, and whenever possible, 
in the students* own classroom. 

ihe size of the experimental discussion croups posed another problem. Teachers 
typically conduct whole class discussions, involving perhaps tventy to thirty 
students, however, research on ;;roup size (Gall and Gall, 1976) suggests that 
the optimal discussion group is five or j,ix students. The research team decided 
to create an optimal context in which to evaluate the effects of questioning skills. 
Therefore, each treatment group consisted of six students. These groups can be 
viewed as representative of recommended teaching practice rather than as represen- 
tative of t ypical teaching practice. 

Fidelity ot treatment with respect to instructional setting is relatively easy 
to accomplish and to monitor. In the experiments on questioning skills, the only 
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problem of note wus t uit liit research team occasionally experienced dilficulty in 
locatini.; .sufficient room for eacr* of tUc treatment groups to conduct their designated 
activity. 

In a curriculum unit iatxtinm two weeks, teachers nii^'ht adrunister one or two tests 
at the end of the unit to measure learniuc. It is difficult for researchers to 
conform to this aspect of classroom practice, since they usually wish to administer 
a variety of tests to measure the many possible effects of teaching, skills. In 
the e:vperiuents on questiouink; skills ten different measures were administered. Six 
of these measures were administered twice^ that is, immediately before and after 
the curriculum unit. One of the measures was administered three .times. 

clearly this amount ut testing is low in ecological validj/ty. Yet it appears necessary 
in order to obtain useful information about possible sttufent effects. To minimize 
the negative impact of mulcinle tests, a number of steps v.ere taken: testing 
was extenvieU over several days; testinp periods were kept short, as were the tests 
themselve.-> , and the tests veie printed m an attractive format. Apparently these 
procedures were eftec^lve bince the research team rarely received complaints from 
students or teachers- about the testing procedure. 

Tiie fidelitv issue witu respect to testintr is whether the tests were administered 
according; to the standardized testing procedures developed by the researchers. 
lUe experimental teachers were speciaUv trained in the instructions for adminis- 
tering: ^^ch test. Also, their performance durinr the actual test administration 
was monitored. Students wtiO were absent at the time of ^roup testing were admin- 
istered tae tests indiviuuallv to preserve completeness of data. These procedures 
resulted in hi^'.h fidelity of testing procedures, that is, there was cl^se correspon- 
dence between the researchers' intent and what actually occurred during data 
collection. 

/ 

Lxperimental control m this tvpe of research requires that the tests be high in 
content validity, iiy content validity is meant the extent to which each test 
reflects the content (information) taught in the curriculum materials and in the 
teaching, episodes. If a test is low in content validity, it will not be sensitive 
to the effects of the teaciiing skill under investigation (Gall, 1974). In the 
experiments on questioning skills, it was possible to achieve high content validity 
because tne content in the curriculum materials and discussions was prespecif ied , 
and theretore could be made available to che test developers. 

Concluding Remarks 

The preceding discussion stressed three methodological requirements that should 
be met in desi>,ninK each aspect of the experimental coatext in research on teaching: 
ecological validity, fidelity of treatment, and experimental .control. The author's 
experiments on questioning; i>kills provide examples of how these requirements can be 
met with respect to each type of context variable. 

A major problem with the experiments is that they were very expensive to conduct. 
Resources usually are not available to design and run teacliing skill experiments on 
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this scale, rowever, ihe na'jor I^osTb iu these experiments wer^ in the design anJ 
development of an instrut.t ional runtext ; scripted disMissiontt, curriculum materials , 
and tei>ts. lliese costs can cllninated or reduced (n future research l)V dcslRninp, 
experLuents wl\Lch use iiistru^ l io::al contexts Jeveloicii li; : re^'ious studies* 



Footnotes 

^Paper presented at tiie annual meeting of the American Lducation Research 
Association, San Francis/o, April 197h, as part of tiie symposium, "\;hat does 
recent research tell us about the study of teaching skills?" I wish to extend 
appreciation to Dr. lieatrice Ward, who organized the symposium and who suggested 
tlie topic of this paper. 

""The work on which this i>aper is based was completed while the author was a 
staff member of the Program for Lffective Education at the Far West Laboratory 
for Lducational Research and Development. 

^The two experiments each incorporated the two-week cirri, ulum unit; hence, the 
four-week total. 
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